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(57) Abstract 

In a'telemantpulation system (12) for manipulating objects located in a workspace at a remote worksite by an operator (14), such as in 
a remote surgical system, the remote worksite having a manipulator with an end effector (39) for manipulating an object at the workspace, 
a controller including a hand control (22) at the operator's station for remote control of the manipulator, an image capture device (19), and 
image output device (20) for reproducing a viewable real-time image, the improvement wherein a position sensor associated with the image 
capture device (19) senses position relative to the end effector (39) and a processor (23) transforms the viewable real-time image into a 
perspective' image with correlated manipulation of the end effector (39) by the hand controller (22) such that the operator can manipulate 
the end effector and the manipulator as if viewing the workspace in true presence. Image transformation according to the invention includes 
translation, notation (25) and perspective correction (29). 
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METHOD FOR TELEMANIPULATION WITH TELEPRESENCE 

BACKGROUND OF THE INVENTION 
This invention relates to telemanipulation using 
telepresence, and more particularly to applications of 
telemanipulation to laparoscopic surgery. 

A telemanipulation system allows an operator to 
manipulate objects located in a workspace from a remote 
control operator's station. For example, in a laparoscopic 
abdominal surgery procedure, the patient's abdomen is 
insufflated with gas, and cannulas are passed through small 
incisions to provide entry ports for laparoscopic surgical 
instruments. Laparoscopic surgical instruments include an 
image capture means for viewing the surgical field and working 
tools, such as forceps or scissors. The working tools are 
similar to those used in open surgery, except that the working 
end of each tool is separated from its handle by an extension 
tube. The surgeon performs surgery by sliding the instruments 
through the cannulas and manipulating them inside the abdomen 
while referencing a displayed image of the interior of the 
abdomen. Surgery by telepresence, that is, from a remote 
location by means of remote control of the surgical 
instruments, is a next step. A surgeon is ideally able to 
perform surgery through telepresence, which, unlike other 
techniques of remote manipulation, gives the surgeon the 
feeling that he is in direct control of the instruments, even 
though he only has remote control of the instruments and view 
via the displayed image. 

The effectiveness of telepresence derives in great 
measure from the illusion that the remote manipulators are 
perceived by the operator of the system to be emerging from 
the hand control devices located at the remote operator's 
station. If the image capture means, such as a camera or 
laparoscope, are placed in a position with respect to the 
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manipulators that differs significantly from the 
anthropomorphic relationship of the eyes and hands, the 
manipulators will appear to be located away from the 
operator's hand controls. This will cause the manipulators to 
5 move in an awkward manner relative to the viewing position, 
inhibiting the operator's ability to control them with 
dexterity and rapidity. However, it is often unavoidable in 
applications such as laparoscopic surgery to move the 
laparoscope in order to obtain the best possible image of the 

10 abdominal cavity. 

Thus, a technique is needed for providing to the 
operator the sense of direct hand control of the remote 
manipulator, even in the presence of a substantially displaced 
^imaging device, such that the operator feels as if he is 

15 viewing the workspace in true presence. 

SUMMARY OF THE INVENTION 
According to the invention, in a telemanipulation 
system for manipulating objects located in a workspace at a 
remote worksite by an operator at an operator's station, such 
as in a remote surgical system, the remote worksite having a 
manipulator or pair of manipulators each with an end effector 
for manipulating an object at the workspace, such as a body 
-cavity, a controller including a hand control at the control 
25 operator's station for remote control of the manipulators, an 
image capture means, such as a camera, for capturing in real- 
time an image of the workspace, and image producing means for 
reproducing a viewable image with sufficient feedback to give 
"the appearance to the control operator of real-time control 
3 0 over the object at the workspace, the improvement wherein 

means are provided for sensing position of the image capture 
means relative to the end effector and means are provided for 
transforming the viewable real-time image into a perspective 
image with correlated manipulation of the end effector by the 
35 hand control means such that the operator can manipulate the 
end effector and the manipulator as if viewing the workspace 
in substantially true presence. By true presence, it is meant 
that the presentation of an image is a true perspective image 
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simulating the viewpoint of an operator. Image transformation 
according to the invention includes rotation, translation and 
perspective correction. 

The invention will be better understood by reference 
5 to the following detailed description in connection with the 
accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1A is a side view of a remote operator station 
10 and of a worksite station in a telemanipulation system 
according to the present invention. 

Fig. IB is a perspective view of a remote operator 
station and a worksite station adapted for stereoscopic 
viewing in a telemanipulation system according to the present 
15 invention. 

Fig. 2 is a diagrammatic perspective view of a 
specific embodiment of the invention wherein the image capture 
means is centered and normalized relative to the viewpoint of 
an operator using the manipulators. 
20 Fig. 3 is a diagrammatic perspective view of a 

specific embodiment of the invention wherein the image capture, 
means is laterally displaced relative to the viewpoint of an 
operator using the manipulators. 

Fig. 4 is a diagrammatic perspective view of a 
25 specific embodiment of the invention wherein the image capture 
means is at a lower position relative to the viewpoint of an 
operator using the manipulators. 

Fig. 5A is a front elevational view of the lenses of 
a stereoscopic image capture means where the lenses are in a 
30 normalized position relative to the viewpoint of an operator 
using the manipulators. 

Fig. 5B is a front elevational view of the lenses of 
a stereoscopic image capture means where the lenses are 
rotated relative to the viewpoint of an operator using the 
3 5 manipulators. 

Fig. 6A is a top plan view of an image of a remote 
manipulator in a telemanipulation system ^that shows a 
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superimposed stereographic four-point coordinate element prior 
to calibration. 

Fig. 6B is a top plan view of an image of a remote 
manipulator in a telemanipulation system that shows a 
superimposed stereographic four-point coordinate element after 
calibration. 

Fig. 7A is a top plan view of an image of a remote 
manipulator in a telemanipulation system that shows the angle 
of displacement in the horizontal of the image capture means 
relative to the manipulators. 

Fig. 7B is an enlarged view of a portion of Fig. 4A 
that shows the combined effect on the position of the end 
effector of a manipulator after a lateral shift. 

Fig. 8 is a geometric depiction of the image of a 
15 manipulator as a projection of a hand control. 

Fig. 9 is a geometric depiction of the actual 
manipulator whose image is depicted in Fig. 8. 



10 
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DESCRIPTION OF SPECIFIC EMBODIMENTS 
Fig. ia shows a telemanipulation system 10 according 
to the present invention with a remote operator station 12 
where an operator 14 (a surgeon, for example) can perform 
telemanipulation on an object at a worksite station 16. The 
remote station 12 includes a video display 20 for the operator 
25 14 to view the worksite and an apparent workspace 22 where the 
operator 14 carries out the actual manipulations by grasping 
hand control means 24, 26, such as surgical instrument 
handles, which are connected to position sensors 51, 55. The 
worksite station 16 includes an image capture means 19, 
3 0 including a sensor means 21 with camera lens 28 of a camera or 
endoscope and understanding that there is an image capture 
means 19 associated with that lens, which captures an image of 
an object located in the workspace 30. (An endoscope would be 
within the body cavity.) The manipulators 32, 34 allow 
3 5 manipulation of the object in the workspace 30 correlated with 
a displayed image by use of end effector means 39, 41. 

The image captured at worksite 16 is transmitted 
through a number of stages which present to operator 14 a 
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real-time image of the object in the workspace 30. In 
particular, sensor means 21, including optical image capture 
means 19, provides a view of the manipulators 32, 34 through a 
camera lens 28, passing such image information via path 13 to 
5 an image processor 23. In addition, image sensor position 
information (camera position) may be passed on path 63 to 
coordinate transformer 43. (For a fixed camera, the camera 
position information can be preset) . Further, mechanical 
position sensing means 59, 61 sense the position of 

10 manipulators 32, 34 mechanically, passing on the position 
information to coordinate transformer 43 via path 157. 

The image processor 23 includes a rotation and 
translation means 25, a perspective correction means 29 and a 
calibration means 27. The rotator is for rotating the image 

15 and the translator is for shifting the rotated image. The 

perspective corrector 29 is primarily for magnifying the image 
and may include some tilt correction. The calibrator 27 may 
have various functions, depending upon the type of image 
input. It is in particular used to calibrate the image to a 

20 known reference coordinate system to enable an operator to 

coordinate motions of the hand controls and the manipulators. 
After the image has undergone transformation through one or 
more of these function blocks, the pixel data is passed on 
path 15 to an imager 31 which drives a video display 2 0, which 

25 in this embodiment is a monoscopic device, and data about the 
image is passed on to the coordinate transformer 43, whereby 
any processed image data potentially affecting control of the 
manipulators (e.g. , magnification, rotation, translation) is 
made available for the control of the manipulators. The 

30 details of the processes which may be affected are explained 
hereinbelow, particularly with respect to calibration. 

The coordinate transformer 4 3 is the principal 
processor of position information. Camera position 
information, manipulator position information, and hand 

35 control position information are received and processed 

therein. In particular, the positions of hand controls 24, 26 
are sensed by position sensors 51, 51 and passed via path 47 
to coordinate transformer 43. After transformation and 
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processing in coordinate transformer 43, control information 
is applied to position-following servo 45, which drives and 
controls manipulators 32, 34 with end effectors 39, 41. The 
operation of each of these blocks will be described in further 
5 detail. 

In operation, the camera lens 2 8 captures the image 
of the object in the actual workspace 30 in a specific 
orientation on image capture means 19. The video display 20 
displays this image so that the operator 14 can view the 
10 object as it is manipulated. The operator 14 may then grasp 

hand control means 24, 26 located in the apparent workspace 22 
to carry out the desired manipulations. The hand control 
means 24, 26 at remote station 12 under instruction of the 
position-following servo 45 control the manipulators 32, 34 at 
worksite station 16, which actually manipulate the object in 
workspace 30. The actual workspace 3 0 is thus effectively 
projected back to the remote operator 14 to create the 
illusion that he is reaching and looking directly into it and 
controlling the object located in workspace 30. Properly 
projected, this results in natural and spontaneous control 
motions by the operator 14, even if he is located in an 
another room or another extremely remote location. 

The problems addressed by the present invention 
arise from the situation where the camera lens 28 is not 
placed at the same position in the real workspace 3 0 relative 
to the manipulators 32, 34 as the eyes of the control operator 
viewing the projected image in the -apparent" workspace 22 
relative to the hand control means 24, 26. A solution is 
provided by the present invention. 

The telemanipulation system according to the present 
invention can also be adapted to accommodate stereoscopic 
viewing. Fig. IB shows all the elements of Figs. 1A, with the 
addition of a second camera lens 36 and image capture means 
35. The two camera lenses 28 and 36 can be separated by about 
35 10% which is the same interocular viewing disparity that one 
experiences when viewing a visual field at 40 cm separation. 
The stereo image is displayed on a stereo video display 
monitor 38 (e.g. using an electronically switched polarizer 37 
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over the screen) and viewed through "cross-polarized 
stereoscopic lenses 40, thus offering a natural image to the 
remote operator 14 so that the operator experiences the 
correct visual feedback when reaching and looking directly 
5 into the actual workspace 30 and- directly manipulating the 
object located therein. The details of the system are 
explained hereinafter. 

Fig. 2 is a diagrammatic perspective view of the 
elements of the worksite station 16 in workspace 30 of the 

10 telemanipulation system, showing* features of Fig. 1 which are 
in a control loop. The system an Operator at a remote station 
to manipulate objects located at ! a ! centerpoint 50 in the 
workspace 30. In the monoscopic system, sensor means 21 with 
camera lens 28 and image capture means 19 captures a real-time 

15 image of the object. The operator 14 uses dual hand control 
means 24, 26 to control left manipulator 32 and right 
manipulator 34, respectively, which' allow remote nanipulation 
of the object at the workspace 30. For hand control means 24, 
26 and manipulators 32, 34, there 'His in this example a fixed 

20 pivot point about which bidirectional angular motion can be 

effected, together with a telescbpxc-like extension capability 
for each of the manipulators and haiid controllers. The 
correlation between the hand control means 24, 26 and the 
manipulators 32, 34, combined with'the image captured by the 

25 camera lens 28, provide sufficient: -feedback to give the 

appearance to the control operator of real-time control over 
the object at the workspace (further improvement is possible 
with tactile feedback). Both left manipulator 32 and right 
manipulator 34 are in this example raised 30° with respect to 

30 an arbitrary plane of orientation /-including a centerline axis 
52 of the workspace 30, to simulate a typical positioning of 
an object in the real local workspace 30. 

In operation, camera lens 28 is at the 0° lateral 
position with respect to the centerline axis 52, such that the 

35 camera lens 28 is between left manipulator 32 and right 

manipulator 34. The face of the camera lens 28 is raised at 
for example a 45° angle with respect to the plane containing 
centerline axis 52 and baseline 53. This camera position and 
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orientation is a close approximation to the actual eye 
position with respect to the manipulators 32 and 3 4 and 
represents a base or reference position. The image captured 
by the camera lens 28 appears as if the operator were looking 
at the centerpoint 50 while standing over the manipulators 32 
and 34 with a 45° angle view into the workspace- Both left 
manipulator 32 and right manipulator 34 appear in the bottom 
of the displayed image (proximal to the operator "s hand 
controls) , evoking a strong sense of telepresence, which means 
that the operator senses direct control of manipulators 32 and 
34, allowing control with dexterity and rapidity, particularly 
where there is tactile feedback from the manipulators 32, 34 
to the hand control means 24, 26. 

In a telemanipulation application in which 
15 positioning of elements is difficult due to obstructions, it 
is often necessary to move the camera lens 28 to different 
positions result in a different view of the object at the 
centerpoint 50. Referring to Fig. 3, a diagrammatic 
perspective view of the elements in workspace 30 of the 
2 0 worksite station 16 of a monoscopic telemanipulation system is 
shown in which the camera lens 28 position is rotated by angle 
0 58 laterally in the horizontal plane away from the 
centerline axis 52. After rotation of the camera lens 28, 
left manipulator 32 and right manipulator 34 are still 
25 inclined downward at a 3 0° angle relative to the plane 

containing centerline axis 52 and baseline 53, and the camera 
lens 28 is still positioned at an angle 0 above the plane 
formed by centerline axis 52 and baseline 53. In order to 
evoke a sense of telepresence in the operator similar to the 
case in which the camera lens 28 is positioned directly over 
manipulators 32 and 34 (as in Fig. 2), according to the 
invention, the captured image projected by the camera lens 28 
is rotated about visual axis 54 through the center of the 
camera lens 28. This compensates for rotation about 
35 "vertical" axis U to effect a static reorientation of the 
apparent manipulator positions. 

It should be understood that camera lens 28 and 
image capture means 19 enjoy a full range of rotation about 
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vertical axis U, and that the angles relative to reference 
planes and the like of the manipulators and the camera are 
dictated by the constraints of the operating environment. 
Additionally, camera lens 28 may be positioned at different 
5 angles relative to the plane formed by centerline axis 52 and 
baseline 53. For example, Fig. 4 shows camera lens 2 8 
positioned at an elevation of 15° above the (arbitrary) 
reference plane formed by centerline axis 52 and baseline 53. 
In this alignment, camera lens 28 is below manipulators 32, 
10 34. 

If the image is purely monoscopic as depicted in 
Fig. 1A, the system can effect static reorientation of the 
manipulators 32 and 34 about an axis 54 through a point, 
specifically center point 50, by rotating the digital image 

15 through rotation means 25. Fig. 3 shows the relevant angles 
of rotation. Angle * 56 denotes the angle of declination of 
the visual axis 54 of camera lens 28 below vertical axis U. 
Angle e 58 denotes the rotation of camera lens 28 position in 
the horizontal plane (formed by lines 52, 53) away from 

20 centerline axis 52 relative to the centerpoint 50. 

Rotation means 25 effects static realignment of the 
manipulators by rotating the real-time image pixel-by-pixel by 
an angle approximately equal to -6, according to known 
methods. After this operation is complete, the left 

25 manipulator 3 2 and right manipulator 34 appear in the bottom 
of the displayed image (lower half of the projected screen). 
The camera lens 28 remains stationary, and the displayed image 
is rotated through image manipulation. Note that if hand 
control means 24, 2 6 at the operator's station are positioned 

30 above the viewpoint of the control operator, the rotation of 
the displayed image will correct the displayed image to the 
point where the manipulators appear in the top of the 
displayed image (upper half of the projected screen) . In 
either case, the transformation of the displayed image allows 

3 5 the operator to view the manipulators as if emerging from the 
operator's hand controls. The remapping of the image is 
effected before actual control can be effected. 
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In addition to effecting static realignment through 
digital image transformation, transformation means 25 may 
effect dynamic synchronization of apparent manipulator tip 
positions with hand control positions by performing the 
5 following coordinate transformation on the video image data. 
The actual position of the manipulator tips in the workspace 
3 0 can be transformed to an apparent position in the displayed 
image so that the manipulators will appear to move as though 
rigidly connected to the operators hand controls. The 

10 altering of the apparent position of the manipulator tips 

improves the dexterity of the operator in handling the object 
in the workspace 30. Because the end point of the end 
effector of the manipulator is known, the point (a,b,c) can be 
related to the angular position and length of the manipulator, 

15 and the point (p,q,r) can be related to the same parameters 
relative to the hand control using well-known trigonometric 
relationships between vectors and their endpoints. Thus: 



20 



cose 1 sine 1 0 
-sine 1 cose* 0 
0 0 1 



cos* 0 sin$ 

0 10 
-sin$ 0 cos* 



cose sine 0 
—sine cose o 
0 0 1 



(Eq. 1) 

In connection with the transformation associated with the 
above equation, the image is rotated by an angle e' selected 

25 by the operator to bring the apparent position of the 

manipulators into substantial registration with the hand 
controls. It is an observation that angle e 1 a -e. This 
transformation describes the relationship between the position 
of the point represented by the end effector means at (a,b,c) 

30 (for either end effector means) relative to the point (p,q,r) 
of the corresponding tip of the manipulator in the apparent 
workspace in the displayed image on video display 20. 

Another method of achieving static reorientation of 
manipulator positions is to rotate the image capture means 

35 about its visual axis. Referring again to the monoscopic 
system depicted in Fig. 1A and Fig. 3, camera lens 28 is 
rotated about its own visual axis 54, an axis normal to the 
plane of the camera lens 28, to the point where left 
manipulator 32 and right manipulator 34 appear in the bottom 
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of the displayed image (lower half of the projected screen), 
iote again that if hand control means at the operator's 
istation are positioned above the viewpoint of the control 
.operator, the rotation of camera lens 28 and image capture 
5 means 19 will correct the displayed image to the point where 
.the manipulators appear in the top of the displayed image 
(upper half of the projected screen) . 

To preserve the stereoscopic effect, in the case of 
^stereoscopic imaging, as depicted in Fig. IB, rotations cannot 
10 be done about separate axes through each camera lens, but 
(referring to Fig. 5A and Fig. 5B) must be done in concert 
about a single axis offset from either lens. Specifically, 
rotation is done normal to center axis 57 passing through the 
.centerpoint 50 and an arbitrary point on center axis 57 
15 between the stereoscopic camera lenses 28 and 36 (Fig. 5A & 

*5B) . This axis is similar to the visual axis 54 described in 
^connection with Fig. 2. Referring to Fig. 5A, the lenses of a 
stereoscopic device are shown in their initial position. 
Center axis 57 shows the fixed relation of each lens of the 

2 0 camera pair and is parallel to a reference axis 59 parallel to 

;an axis in the plane formed by manipulators 32, 34 
intersecting at the centerpoint 50, where the axis is normal 
to a line bisecting the manipulators and passing through the 
centerpoint 50. In order to reorient the displayed image 
25 through rotation of the image capture means, center axis 57 is 
/canted relative to a reference plane 59 passing through 
ienterpoint 50, which plane includes reference axis 59, as 
shown in Fig. 5B. 

■ ■P- There is a limitation on the amount of visually 

3 0 acceptable rotation of the stereoscopic image capture means 

19, 35 and the elevation of the image capture means 19, 35 
^relative to the plane of the manipulators 32, 34. The 
"elevation cannot be so great as to make it impossible to 
/change the relative view angle of each of the two manipulators 
35 -relative to one another. Clearly, if angle * equals 90° 

/elevation (where the viewing axis 54 lies in the reference 
^plane formed by lines 52 and 53), no useful change in the 
/relative view angle will be achieved by rotating the image. 
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At other angles of elevation, the limitation depends on the 
separation angle of the manipulators 32, 34 and secondarily on 
the separation of the stereoscopic lenses 28 , 36. 

In addition to achieving static reorientation of 
manipulator positions by rotation of the camera lens 28, the 
system can effect a dynamic realignment by performing a 
coordinate transformation through translation means 25. The 
actual position of the manipulator tips in the workspace 30 
can be transformed to an apparent position in the displayed 
image so that the manipulators will appear to move as though 
rigidly connected to the operator's hand controls. The 
altering of the apparent position of the manipulator tips 
improves the dexterity of the operator in handling the object 
in the workspace 30. 

Fig. 8 and Fig. 9 depict the image 132 of a 
manipulator (32) and an actual manipulator 32, respectively, 
relative to a hand control 24. In this example, and comparing 
Fig. 2, manipulators and corresponding controllers represented 
by hand controls are of a type utilizing a single pivot point 
151, 161 in connection with the position sensors 51, 61 with 
two dimensional pivot about the point (s) and extension along 
the axis of the manipulator 32. Other motions consistent with 
these actuations, such as longitudinal rotation of the 
manipulator about its axis is contemplated by the invention. 
25 With reference to Fig. 8 and Fig. 9, movement of the hand 
control 24 causes the manipulator tip 72 to move to a new 
point (a,b,c) in workspace 30 such that the image 132 of the 
manipulator tip 172 moves to a new point (p,q,r) in the 
apparent workspace 22, that point appearing in the view of the 
operator to be at the extended end of the hand control 24. it 
may be preferable to express the servo commands relating the 
hand control and the corresponding manipulator in their polar 
coordinate systems (n, ^ , L) and (n», tf« , L 1 ) . These polar 
coordinates and their respective points in Cartesian 
coordinate systems are related by well-known polar to 
Cartesian transformations. 

Referring again to Fig. 3, the specific angles of 
rotation used in calculating the coordinate transformation are 



20 



30 



35 



<WO 9S30964A1 J_> 



WO 95/30964 



PCT/US95/05669 



13 



10 



15 



shown. Angle * 56 denotes the angle of declination of the 
visual axis 54 of camera lens 2 8 below vertical axis U. Angle 
e 58 denotes the rotation of camera lens 28 in the horizontal 
plane away from centerline axis 52 relative to the centerpoint 
50 in the workspace 30. Angle r 60 denotes the rotation of 
camera lens 28 about its visual axis 54 . 

In operation of a monoscopic telemanipulation 
system, camera lens 28 and image capture means 19 are rotated 
about visual axis 54 as described above. The coordinates 
(a,b,c) in a reference orthogonal Cartesian coordinate system 
of the three-dimensional workspace 3 0 define the actual 
position of the tip of a manipulator, such as left manipulator 
32* The following matrix equation relates the desired 
apparent position (p,q,r in orthogonal Cartesian space) of the 
manipulator tip in the displayed image in video display 20 to 
the actual position (a,b,c) of the manipulator tip in the 
workspace 30: 
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cosr sinr 0 
-sinr cosr 0 
0 0 1 



cos* 0 sin* 
0 10 
-sin* 0 cos* 



cose sine 0 
—sine cose 0 
0 0 1 



(Eq. 2) 

When the manipulator tip is displayed at a position (p,q,r) in 
the displayed image in video display 20, the manipulator will 
appear to the operator as if it is actually at the end of the 
operator^ rigid hand control device. The coordinate 
transformation improves the ease with which the operator can 
handle objects in the workspace using a telemanipulation 
system. 

In the case of stereoscopic imaging, the stereo 
image capture means 19, 35 is rotated relative to a reference 
axis 59 parallel to an axis in the plane formed by 
manipulators 32, 34 intersecting at the centerpoint 50, where 
the axis is normal to a line bisecting the manipulators and 
passing through the centerpoint 50, as shown in Fig. 5B. 
Angle r 60 measures the amount of rotation of the stereoscopic 
lenses 28, 36, and its value can be used in Eq. 2 to calculate 
the proper coordinate transformation for stereoscopic viewing. 
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..V". In order to ensure that the movements of the 

^manipulators 32, 2 4 in workspace 3 0 properly track the 
anovements of the hand controls 24, 2 6 in the operator's 
■apparent workspace 22 even without complete knowledge of all 
5 angles and positions, the operator can establish a calibration 
reference for manipulators 32, 34 as they are viewed in the 
displayed image in video display 2 0 in connection with the 
Jposition-following servo. Referring to Fig. 6A f which shows 
j£he image displayed in video display 20, a four-point 
10 coordinate graphic element 62 for example in the form of a 

tetrahedron or cube structure in three-dimensional view may be 
^superimposed in three-dimensional space on the stereo image 
^display, providing a coordinate reference in the three- 
dimensional image space. To calibrate the position of a 
15 single manipulator with respect to its corresponding hand 

^control, the system "opens" the control loop, and the operator 
•il4 moves hand control 24, for example, while observing the 
motion of the tip of manipulator 32, steering the tip until it 
appears to be touching a first reference point 64 of 
20 superimposed graphic element 62, as shown in Fig. 6B. (Since 
;the motion of the hand control and manipulator tip have not 
yet been coordinated, the alignment of the tip with the first 
preference point may require very deliberate effort.) The 
operator 14 then indicates to the system that superposition of 
25 ^manipulator and reference point has been achieved (e.g., a 
^set" signal is sent to the system) . 

grp The system then locks the manipulator 32 into place, 

;1opens the control loop by decoupling it from the hand control 
and inst ructs the operator 14 to release the hand control 
30 ^4. The system adjusts the extension L (Fig. 8 and 9) of the 
-.hand control to match that of the manipulator L 1 , by inserting 

offset a 3 = L ~ L */ so that when the control loop is 
l? losed ' there will be no reactive motion by either device. 
v§E£ at is ' tne apparent extension positions of the hand control 

and manipulator 32 must be identical when compared in the 
gsbontrol loop. The system then closes the control loop and 
>gunlocks the manipulator 32 , returning control to the operator 
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The operator then moves the hand control about its 
pivot point to an angular orientation at which the 

operator senses that the image of the manipulator appears to 
emerge from the operator's hand control. Similar to the 
process described above, the system computes transformations 
which ensure that there will be no reactive motion by either 
master or slave when the control loop is closed. The system 
calculates angular offsets o x = ^ - and a 2 = n - n* and 
transforming the apparent position of the master or the slave 
prior to closing the control loop. The system now records the 
positions in three-dimensional space of the hand control 
master (^ lt n lf L a ) and the manipulator slave W lt fl' 1# L^). 

The operator repeats the elements of this process 

with the remaining reference points of the superimposed 

graphic element 62. The system may then derive and install 

the following linearized equation relating incremental changes 

in the position of the hand control masters 24, 26 to 

incremental changes in the position of the manipulator slaves 

32 , 34, using the data sets to determine the coefficients of 

the equations relating the positions: 

An 1 = k ia An + k 12 A* + * 13 AL 
AS" = k 21 An + k 22 A* + ^23^ 
AL ' = k 31 An + k 32 A* + k 33 AL 

The solution to the above linearized equation is as follows: 



k ll k 12 k 13 
k 21 k 22 k 23 



k 31 k 



32 k 33 



An 1 } An 1 2 An» 3 

A* 1 ! A** 2 A*" 3 
AL'i AL' 2 AL* 3 



AH} An 2 An 3 

A*! £& 2 A *3 
AL^ AL 2 AL 3 



-1 



(Eq. 3) 

The system installs these coefficient values in the coordinate 
transformer 43 which controls servo 45, with appropriate 



offsets a 



l' 



and a 



3' 



so that there is no reactive motion 



when the loop is closed. 

In an alternative embodiment, calibration of the 
manipulators is achieved through virtual movement with the 
assistance of the system. Referring to Fig. 6B, the system 
moves manipulator tip 32, rather than the operator guiding the 
manipulator tip 32, to one of four defined points in the 
three-dimensional workspace 30, such as reference point 64 as 
seen by the operator. Using the hand control 24, the operator 
14 then dynamically calibrates the position of the manipulator 
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32 by steering an overlaid graphic dot until it appears 
superimposed on the manipulator tip 32. The operator 14 then 
indicates to the system that superposition of manipulator tip 
3 2 and reference point 64 has been achieved, and the 
coordinates of the manipulator 32 and hand control 2 4 are 
recorded. The process is repeated for the remaining reference 
points, after which the system derives and installs a 
coordinate transformation formula in the coordinate 
transformer 43, as described in the above embodiment. 

In actual practice, it is preferable for the 
surgeon, rather than the system, to initiate the calibration 
process if the invention is being used in laparoscopic 
surgery. During surgery, the calibration process is being 
carried out within a patient's abdomen, where there is little 
15 room to maneuver. Hence, automatic movements of the 

manipulator, however small, may be considered less desirable 
than operator-controlled movements. 

Another method for evoking a sense of telepresence 
in a telemanipulation system involves the use of a specific 
coordinate transformation to compensate for other changes in 
the displayed image, such as a lateral shift or a scale 
change. The camera may undergo a lateral or angular 
displacement, causing the displayed image to shift. in 
addition, the camera may be capable of magnifying the object 
in the workspace, which causes a scale change and a 
displacement of the apparent pivot point of the manipulator. 

Figs. 7A and 7B show the combined effect of a 
lateral shift of the image and a scale change brought about by 
magnification of the image. Fig. 7 A shows a portion of the 
displayed image, including a manipulator 32, in a two- 
dimensional field. The center of the image is at coordinates 
(0,0). The operator experiences the best possible sense of 
telepresence if the manipulator tip 72 at coordinates (u,v) in 
the image field appears to move as if it were rigidly attached 
35 to the control device in the operator's hand. The control 
device is pivoted at point (m,n) in the figure. The 
manipulator lies at an angle a 74 to the y-axis, and the 
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distance from pivot point (m,n) to manipulator tip (u,v) is 
length L 76. 

Fig. 7B shows what the operator would see if the 
image were magnified by a factor M. The center of the image 
5 is shifted laterally by a distance of Ax and Ay, and the new 
apparent coordinates of the manipulator tip 72 are (u'.v 1 ). 
In order to ensure a desired level of telepresence, angle a 74 
and length L 76 are remapped through perspective correction 
means 29 in the displayed image to give the operator the 
10 impression that the manipulator tip 72 is still rigidly 

attached to the hand control device. The following pair of 
equations describe the remapping of angle a 74 into angle a* 
78 and length L 76 into length L 1 80: 

a* = arctan t(u' - m) / (V - n) ] and 
15 L ' = [(u* - m) 2 + (v* - n) 2 ] 1 * 

where : 

u ' = M(u - Ax) v ' = M(v - Ay) and where 

u = L(sina) + m v = L(cosa) + n 

When a and L are remapped according to the above equations, 
20 the manipulator tip 72 appears in the displayed image to move 

as if it were rigidly connected to the operator's hand control 

device. 

The above relationships can be extended to include 
transformations in three dimensions in order to compensate for 

25 displacement of the manipulators when the camera lens 28 is 
rotated about its own visual axis 54 , as in the embodiment 
described with respect to Fig. 3. In all cases, the desired 
goal of maintaining the perceived plane containing the two 
manipulators coincident with the plane of the two hand 

30 controls is achieved. 

The invention has now been explained with reference 
to specific embodiments. Other embodiments will be apparent 
to those of ordinary skill in the art upon reference to the 
present description. For example, the invention can be 

35 extended to articulated manipulators with multiple points of 

rotation and translation or with pivot points at locations not 
physically attached to the manipulators- It is therefore not 
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intended that this invention be limited, except as indicated 
by the appended claims. 
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WHAT IS CLAIMED IS : 

1 1. In a telemanipulation system for manipulating 

2 objects located in a workspace at ..a-, remote worksite by a 

3 control operator from a control operator station, said 

4 telemanipulation system including manipulator means at the 

5 worksite with end effector means for manipulation of an object 

6 in the workspace, controller means including hand control 

7 means at the control operator station for remote control by 

8 the control operator of the manipulator means, image capture 

9 means for taking a real-time image of the workspace, and image 

10 producing means for producing a display on a monitor for eyes 

11 of the control operator, said controller means further 

12 comprising: 

13 means coupled to said -image capture means for 

14 processing the real-time image into a processed image for 

15 display to the control operator wherein said image capture 

16 means is at an angle and a positiotv-relative to the end 

17 effector means which differs from! a relative position of eyes 

18 of the control operator and the hand control means, said image 

19 processing means correlating image^of said end effector means 

20 with said hand control means. 

1 2. The system according to claim 1 wherein said 

2 image processing means further comprises: 

3 means for converting an- actual position of the end 

4 effector means in the workspace tc!^a perceived position of the 

5 end effector means in the viewable real-time image, the image 

6 capture means being monoscopic and being positioned at an 

7 angle e on a reference plane, said"~image capture means being 

8 above the reference plane in a reference space, and wherein 

9 the eyes of the control operator are positioned at an angle 0° 

10 relative to a base line on said reference plane, said eyes 

11 being above said reference plane in said reference space, the 

12 viewable real-time image being rotated pixel-by-pixel by an 

13 angle e 1 , which is approximately equal to -e. 
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1 3. The system of claim 2, wherein coordinates of 

2 said perceived position of the end effector means are 

3 dynamically related to coordinates of said actual position of 

4 the end effector means by the following relationship: 



10 



cos* o sin* 

0 10 
-sin* 0 cos* 



cose sine o 
-sine cose o 
0 0 1 



5 p cose 1 sine 1 o 

6 q = -sine' cose* 0 

7 r p o l 

8 where 

9 [P#<3/r3 are coordinates of said perceived position, 
[a,b,c] are coordinates of said actual position, 

11 angle e is the rotation of the image capture means in a 

12 horizontal plane about a vertical axis, 

13 angle e* being substantially equal to -e, and 

14 angle * is the angle of declination below said vertical 

15 axis of an axis normal to a surface of the image capture 

16 means. 

1 4. The system according to claim 1 wherein said 

2 image procesing means further comprises: 

3 means for static realignment of the manipulator 

4 means effected by rotating the image capture means about an 

5 axis normal to a surface of the image capture means to a 

6 position where each manipulator means appears in a viewable 

7 real-time image as presented to the control operator, where 

8 the hand controls are below a viewpoint of the control 

9 operator . 

1 . 5. The system according to claim l wherein said 

2 image processing means further comprises: 

3 means for static realignment of the manipulator 
means effected by rotating the image capture means about an 
axis normal to a surface of the image capture means to a 
position where the manipulator means appear in the viewable 
real-time image as presented to the control operator, where 

8 the hand controls are above a viewpoint of the control 

9 operator. 
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1 6. The system according to claim 4 wherein the 

2 image capture means comprises: 

3 a monoscopic camera, wherein rotation of the image 

4 capture means is about a visual axis of said monoscopic camera 
through a center of a lens of said monoscopic camera. 

1 7 . The system according to claim 4 wherein the 

2 image capture means comprises: 

3 a stereoscopic dual aperture camera with a normal 

4 axis bisecting a line between each aperture of said 

5 stereoscopic camera, wherein rotation of the image capture 

6 means is about a visual axis of said stereoscopic camera 

7 through the center of said normal axis. 

1 8 . The system according to claim 4 wherein said 

2 image processing means further comprises: 

3 mtans for translating an actual position of the end 

4 effector means in the workspace to a perceived position of the 

5 end effector means in the viewable real-time image such that 

6 motion of the end effector means appears rigidly connected to 

7 the hand control means. 

1 9. The system according to claim 8 wherein the 

2 translating of said actual position to said perceived position 

3 is according to the following relationship: 

4 p cosr sinr 0 cos* 0 sin* cose sine 0 a 

5 q = -sinr cosr 0 0 10 -sine cose 0 b 

6 r 0 0 1 -sin* 0 cos* 0 0 1 c 

7 where 

8 [P/<3*r] are coordinates of said perceived position, 

9 [a,b,c] are coordinates of said actual position, 

10 angle r is the rotation of the image capture means about 

11 an axis normal to a surface of the image capture means, 

12 angle * is the angle of declination of said normal axis 

13 below a vertical axis, and 

14 angle e is the rotation of the image capture means in a 

15 horizontal plane about said vertical axis. 
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1 10. For use with a telemanipulation system for 

2 manipulating objects located in a workspace at a remote 

3 worksite by a control operator from a control operator 

4 station, said telemanipulation system; including a manipulator 

5 means at the worksite with end effector means for manipulation 

6 of an object in the workspace, controller means including hand 

7 control means at the control operator station for remote 

8 control by the control operator of the manipulator means, 

9 image capture means for taking a xeal-time image of the 

10 workspace, and image producing means for reproducing a 

11 viewable real-time image on a monitor for eyes of the control 

12 operator, a method for dynamic calibration of the manipulator 

13 means through actual movement usirig a control loop wherein 

14 movements of the manipulator means directly follow movements 

15 of the hand control means, the method comprising the steps of: 

16 a) superimposing a representation of a four-point 

17 coordinate graphic element in three.-dimensional space on the 

18 viewable real-time image; !: , !; 

19 *>) causing the hand control means to move to 

20 control motion of the end ef f ector, f means ; thereupon 

21 c ) positioning the end; ; ef £ ector means so that the 

22 end effector means appear to be touching a first reference 

23 point of said graphic element in the viewable real-time image; 

24 d ) indicating to the system that superposition of 

25 the end effector means and said f iorst reference point has been 

26 achieved; 

27 e) determining an offset by the relation a 3 = L» - 

28 l to adjust the extension L of the hand control means to match 

29 the extension L» of the manipulator ; means; 

30 f ) causing the hand control means to move about its 

31 pivot point to an angular orientation (^,n) , where the 

32 manipulator means in the viewable r.eal-time image appears to 

33 emerge from the hand control means ;i, : 

34 g) determining offsets according to the 

35 relationships a 1 = * - and o 2 - n* for adjusting the 

36 angular orientation (*,n) of the haiid, control means to match 

37 the angular orientation pi",!} 1 ) of the manipulator means; 
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38 h) recording a control position (^,n,L) in three- 

39 dimensional space of said hand control means; 

4 0 i) recording a controlled position (¥• ,0' ,X' ) in 

41 three-dimensional space of said manipulator means 

4 2 corresponding to the control position; and 

4 3 j) repeating steps a) through i) for each remaining 

44 reference point, thereby establishing calibration. 
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11. The method of claim 10 further comprising the 

step of synchronizing incremental changes in position of the 

hand control means to incremental changes in position of the 

manipulator means according to the following set of equations: 

Art' as k xl An + k 12 A* + k 13 AL 
A¥" as k 21 AIl + k 22 A* + k 23 AL 
AL' ~ k 31 An + k 32 A* + k 33 AL 



whose solution is as follows: 



k ll k l2 k l3 
k 21 k 22 k 23 

k 31 k 32 k 33 



An' x An» 2 An* 3 



A* 1 ! A* 2 
AL 1 x AL' 2 



A*- 3 
AL 1 



An, An, Afi- 



A* 2 
ALi 



A* 2 
AL 2 



A* 3 
AL 3 



-1 



1 12 . For use with a telemanipulation system for 

2 manipulating objects located in a workspace at a remote 

3 worksite by a control operator from a control operator 

4 station, said telemanipulation system including a manipulator 

5 means at the worksite with end effector means for manipulation 

6 of an object in the workspace, controller means including hand 

7 control means at the control operator station for remote 

8 control by the control operator of the manipulator means, 

9 image capture means for taking a real-time image of the 

10 workspace, and image producing means for reproducing a 

11 viewable real-time image on a monitor for eyes of the control 

12 operator, a method for dynamic calibration of the manipulator 

13 means through virtual movement using a control loop where 

14 movements of the manipulator means directly follow movements 

15 of the hand control means, the method comprising the steps of: 

1 a) causing the end effector means to move to a first 

2 reference point of a selection of four points distributed in 

3 three-dimensional space on the viewable real-time image ; 

4 b) manually moving the hand control means so that an 

5 overlaid graphic dot appears superimposed on the viewable 
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24 

image of the end effector means and said first reference 
point; 

c) indicating to the system that superposition of 
the end effector means and said first reference point has been 
achieved; 

d) calculating an offset a 3 = L' - L to adjust the 
extension L of the hand control means to match the extension 
L 1 of the manipulator means; 

e) moving the hand control means about its pivot 
point to an angular orientation (*,n) , where the manipulator 
means in the viewable real-time image appears to emerge from 
the hand control means; 

f) calculating offsets = * - and o 2 = n - n 1 
to adjust the angular orientation (*,n) of the hand control 
means to match the angular orientation (¥' f n*) of the 
manipulator means; 

g) recording a position (*,n,L) in three-dimensional 
space of said hand control means; 

h) recording a position (^n^L') in three- 
dimensional space of said manipulator means; and 

i) repeating steps a) through h) for each remaining 
reference point, thereby establishing calibration, 
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13. The method of claim 12 further comprising the 
step of relating incremental changes in position of the hand 
control means to incremental changes in position of the 
manipulator means according to the following set of equations; 



An* a k 1:L An + k 12 i 

+ k 22 A* + 
AL» a k^Aft + k 



A*' s k 21 Aft 



8 whose solution is as follows: 



32 



►A* + 

;a* 
;a* + 



k 13 AL 
k 23 AL 
k 33 AL 



J" 


*12 


*13 




x 21 


k 22 


k 23 




k 31 


*32 


*33 





An*! An« 2 



A* ! 

AL * 



A*' 



1 ^2 



An» 3 

A¥' 3 
AL' i 



Aft, 
A¥ 3 
AL-, 



An 2 

A¥ 2 
AL 2 



Aft 3 
A¥ 3 
AL 3 
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14. In a telemanipulation system for manipulating 
objects located in a workspace at a remote worksite by a 
control operator from a control operator station, said 
telemanipulation system including a manipulator means at the 
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5 remote worksite with end effector means for manipulation of an 

6 object in the workspace, controller means including hand 

7 control means at the control operator station for remote 

8 control by the control operator of the manipulator means, 

9 image capture means for taking a real-time image of the 

10 workspace, and image producing means for producing a display 

11 on a monitor for eyes of the control operator, control means 

12 comprising: 

13 means coupled to said image capture means for 

14 processing the real-time into a processed image for display to 

15 the control operator wherein said image capture means is at an 

16 angle and a position relative to the end effector means which 

17 differs from a relative position of eyes of the control 

18 operator and the hand control means, and which further differs 

19 in scale and in lateral position, said image processing means 
2 0 correlating manipulation of said end effector means with said 
21 hand control means, said image processing means further 

2 2 comprising: 

23 means for performing a perspective correction in the 

24 viewable real-time image following a lateral shift and a scale 

25 change of the image capture means. 

1 15 . The system according to claim 14 wherein said 

2 perspective correction means comprises means for performing 

3 transformations according to the following relationships: 

4 a' = arctan [ (x* - m) / (y' - n) ] , 

5 L ' = [(x* - m) 2 + (y» - n) 2 ]* 5 , 

6 x 1 = M(x - Ax) , 

7 y" = M(y - Ay) , 

8 x = L(sina) + m, 

9 y = L(cosa) + n 

10 where 

11 coordinates (m,n) describe a position of a pivot 

12 point of the manipulator means before said lateral shift and 

13 said scale change occur, 

14 coordinates (m^n 1 ) describe a position of said 

15 pivot point of the manipulator means after said lateral shift 

16 and said scale change occur, 



BNSDOCID < WO 953096AA 1 _l_> 



WO 95/30964 



PCT/US95/05669 



26 

17 \V coordinates (x,y) describe a position of the end 

18 reffector means before said lateral shift and said scale change 

19 ;occur, 

20 coordinates (x',y'J describe a position. of the end 

21 effector means after said lateral shift and said scale change 

22 occur, 

-; coordinates (Ax, Ay) describe the change in position 

the end effector means brought about by said lateral shift, 
25 4r factor M describes the magnification factor of said 

2 6 ^scale change, 

27 angle a is the rotation of the manipulator means 

28 about a vertical axis before said lateral shift and said scale 

29 change occur, 

angle a • is the rotation of the manipulator means 
about said vertical axis after said lateral shift and said 
32 scale change occur, 

fr4\ length L is a distance between said pivot point and 

34 the end effector means before said lateral shift and said 

35 scale change occur, and 
g length L* is a distance between said pivot point and 
:the end effector means after said lateral shift and said scale 



23 
24 



30 
31 



36 
37 



38 change occur. 



VV 16 * In a telemanipulation system for manipulating 

objects located in a workspace at a remote worksite by a 
; control operator from a control operator station, said 
.telemanipulation system including manipulator means at the 
worksite with end effector means for manipulation of an object 
in the workspace, controller means including hand control 
means at the control operator station for remote control by 
£he control operator of the manipulator means, image capture 
means for taking a real-time image of the workspace, and image 
producing means for producing a display on a monitor for eyes 
Jof the control operator, said controller means further 
11 Comprising: 

means coupled to said image capture means for 
£" CeSSing the real " tiine image into a processed image for 
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14 display to the control operator wherein said image capture 

15 means is at an angle and a position relative to the end 

16 effector means which differs from a relative position of eyes 

17 of the control operator and the hand control means, said image 

18 processing means correlating image of said end effector means 

19 with said hand control means; 

20 servo control means for controlling position of said 

21 manipulator means; 

22 first position sensing means coupled for sensing 

23 position of said manipulator means in said workspace; 

24 second position sensing means coupled for sensing 

25 position of said hand control means at the operator station; 

26 and 

27 coordinate transformation means coupled to said 

28 image processing means, to said first position sensing means 

29 and to said second position sensing means for transforming 

3 0 information about manipulator position, hand control position 

31 and processed image position and into control signals applied 

32 to said servo control means. 



1 17. The apparatus according to claim 16 further 

2 including: 

3 third position sensing means coupled to said image 

4 capture means for sensing position of said image capture means 

5 at the worksite; and wherein 

6 said coordinate transformation means is coupled to 

7 said third position sensing means for transforming position 

8 information about said image capture means into said control 

9 sicy; a Is. 

1 18. For use with a telemanipulation system for 

2 manipulating objects located in a workspace at a remote 

3 worksite by a control operator from a control operator 

4 station, said telemanipulation system including a manipulator 

5 means at the worksite with end effector means for manipulation 

6 of an object in the workspace, controller means including hand 

7 control means at the control operator station for remote 

8 control by the control operator of the manipulator means, 
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9 image capture means for taking a real-time image of the 

10 workspace, and image producing means for reproducing a 

11 viewable real-time image on a monitor for eyes of the control 

12 operator, a method for dynamic calibration of the manipulator 

13 means through actual movement using a control loop wherein 

14 movements of the manipulator means directly follow movements 

15 of the hand control means, the method comprising the steps of: 

16 superimposing a representation of a four-point 

17 coordinate graphic element in three-dimensional space on the 

18 real-time image; 

19 causing the hand control means to move; thereupon 

20 positioning the end effector means under control of 

21 the hand control means so that the end effector means appears 

22 to be touching reference points of said graphic element in the 

23 real-time image; and 

24 establishing a tracking relationship between the 

25 hand control means and the end effector means, thereby 

26 establishing calibration. 

1 19. For use with a telemanipulation system for 

2 manipulating objects located in a workspace at a remote 

3 worksite by a control operator from a control operator 

4 station, said telemanipulation system including a manipulator 

5 means at the worksite with end effector means for manipulation 

6 of an object in the workspace, controller means including hand 

7 control means at the control operator station for remote 

8 control by the control operator of the manipulator means, 

9 image capture means for taking a real-time image of the 

10 workspace, and image producing means for reproducing a 

11 viewable real-time image on a monitor for eyes of the control 

12 operator, a method for dynamic calibration of the manipulator 

13 means through virtual movement using a control loop where 

14 movements of the manipulator means directly follow movements 

15 of the hand control means, the method comprising the steps of: 

16 automatically causing the end effector means to move 

17 through a preselected sequence of reference points distributed 

18 in three-dimensional space on the viewable real-time image; 
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19 manually moving the hand control means so that an 

20 overlaid graphic dot appears superimposed on the viewable 

21 image of the end effector means at each of said reference 

22 points; 

23 registering position of the hand control means at 

24 the position of each reference point; and 

25 establishing a tracking relationship between the 
2 6 hand control means and the end effector means, thereby 

27 establishing calibration. 
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